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ABSTRACT 

Possible bias due to sampling problems or low 
response rates has been a troubling "nuisance" variable in empirical 
research since seminal and classical studies were done on these 
problems at the beginning of this century. Recent research suggests 
that; (1) earlier views of the alleged bias problem were misleading; 
(2) under a variety of fairly well-specified conditions, allegedly 
biased samples are in fact random; and (3) "de facto" biased samples 
can ard will "jump" from being biased to being random samples. These 
conditions occur when the population is fuzzy and dynamic and when 
the number of factors structuring the population and those 
influencing responding or non-responding are large, interact with 
each other, and have a trivial effect, at best. Researchers and 
theorists may be rejecting and disregarding what might be reasonable 
and highly valid data. Actual data from a number of state-wide 
studies with response rates in the 15% to 30% range illustrate this 
view. A variety of simulation studies is needed to evaluate these 
views on bias. The Catastrophe Theory of non-linear functions, 
family-wise error rates, and fuzzy subset theory are also considered < 
(Author/SLD) 
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Abstrac:^ 

Possible bias due to sampling problems or low response 
rates has been an extremely troublesome "nuisance" variable 
in all areas of empirical research ever since seminal and 
classical research studies were done on these problems at the 
beginning of this century. 

A wide variety of work, data, and new theories in many 
different areas, however, strongly suggest that these 
classical formulations and views of the alleged bias problem 
are essentially incorrect, misleading, and overly simplistic, 
and that under a variety of fairly we ! I -spool f i ed conditions 
allegedly biased samples are in fact random <most probably), 
and that de facto biased samples can and will "Jump" from 
being biased to being random samples, in the manner outlined 
and described by Catastrophe theory <Zeeman, 1976). These 
conditions are when the population is fuzzy and dynamic, and 
the number of factors "structuring" the population <or 
sample) and those influencing responding or non-responding 
are large, interact with each other, and net out to zero, or 
the statistical equivalent o-f zero (namely, a trivial effect 
at best). Researchers and theorists in many areas, 
therefore, may be rejecting and disregarding what might be 
reasonably and highly <if not perfectly) valid data and 
studies. Actual data from a number of recent &tate-wide 
studies of large, complex, and dynamic populations which had 
response rates in the 15X to 30% range empirically support 
this view. 

A wide variety of simulation studies of these newer 
views, which are outlined in this paper, are needed. Some of 
these needed simulation studies, as well as the nature of the 
problem<s>, are also outlined in this paper. 

The very B-l;rong suggestions of these newer views, if they 
prove to be essentially correct upon empirical simulation, 
would be a very important discovery, not only for the areas 
of research methodology and statistics, but also fcr the 
areas of cognitive psychology and machine intelligence. 
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Increasingly in the social sciences, researchers have 
been attempting to conduct important surveys of large and 
complex real-world populationG on critically important issues 
and questions. One problem that has continually plagued 
these research efforts, as well as the evaluation and 
acceptance of findings of these surveys has been the low 
response rates obtained from subjects nominally considered to 
be part of the population being surveyed. Typically, 
response rates to surveys of large, complex, real-world 
populations tend to be 25% to 40% of the sample drawn for 
participation. Typically, the sar.ple drawn tends to be 10% 
to 20% of the nominal population, so that the resulting 
sample of responders tend to constitute data on 5% to 10% of 
the nominal population. 

Usually, these low response rates are most interpreted 
as Indicating that there I s a de facto bias in the analysis 
sample. Consequently, the results of the analyses done are 
usually strongly challenged and deemed unacceptable because 
of this alleged bias and both the data and results of 
analyses are unquestionably considered to be highly dubious 
and suspect at best. The key and fundamental problem, 
however, is that the accusation and claim of bias is only a 
dfi jurg. Inferred, extrapo I r.ted , and "armchair" claim of 
L»as, and is not a de fagto proof of bias <even 
probabilistically), and that the very theory upon which this 
inferential, armchair conclusion is based is rift with deep 
core definitional and conceptual problems, contradictions, 
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impossibi I i ties, and several other major flaws which makes 
this theory extremely dubious at best In terms of providing 
an adequate or actual model of events and what actually 
occurs in the real world (rather than in urns with balls that 
have a single property) and real world situations. 

This fundamental fact <and problem) has led many 
theorists and researchers in all areas of science and social 
science to strongly question and challenge the general 
c I ass i ca I theoret i cal perspective, vi ews and mo dels developed 
primarily in the first half of this century. These views 
have been crumbling and stumbling badly in all areas of 
science and social science In the last half of this century, 
particularly as more and more data emerges that contradict 
this general view, and more conceptual and theoretical 
analyses are done of this view. These analyses have tended to 
produce newer and very different theories rather than 
revisions or modification of the general classical theory, 
which is essentially a strict set theoretic, logical 
positive, linear, one-dimensional view and model of the world 
and its events and operations. 

Prwticing researchers (in particular^) and theorists in 
all areas of science and social science are currently 
extreme I y uncomf ortab le, suspicious of, and ill at ease with 
the classical models, positions, and views as they are daily 
turn i ng out to be extreme over — s i mp I i f i cat i ons that break 
down quickly and do not work (by a long shot) even as a 
"very broad gauge macro model" in the real world of everyday 
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life and reGearch. The real everyday world is qualitatively 
very different (see Cronbach, 1988> from the "world" of the 
the "simple experiment," or the simple "hermetic lab, T-maze 
or urn." Compared to these over simpi if ied "worlds," the 
everyday world of reai life and real experiments is "complex 
and even "highly complex," and this "complexity" is very 
different from classical (over) "simplicity," and not an 
extension of this "simplicity," but rather a new area 
requiring a new science; namely, a "science of complexity" 
(see Pagels, 1988). The real everyday world is not an 
extension of the "simple classical model," but rather, it is 
a quantum leap from their "simplicity," and a "jump" of the 
kind character i st i zed by Catastrophe theory (ileeman, 1976), 
which is a general theory of non-linear phenomena and events 

This great and rapidly emerging "discontinuity" between 
the "simple" and the "complex" has been strongly impacting 
practicing researchers and theorists in all areas of science 
and social science in the past five years, but the impact in 
the area of sampling, statistical and research theory has 
been minimal at best. This emerging discontinuity, however, 
has begun to make some impact on these three areas very 
recently, as several leading theorists have begun to express 
their k^xtreme discomfort with the classical views on key 
topics in these three areas, and their concerns with and 
alienation from classical theories and models in these three 
areaF/ that are biythely and unquest i on i ng I y accepted by the 
majority of professionals working in these three areas. 



Nowhere are these later points more evident than in the area 
of the topic of this paper. 

Angoff <198B) has called the problem of bias the most 
ill and poorly defined, and poorly understood concept in all 
of measurement, statistics, and research, and the on© concept 
most in need of clarification, analysis, conceptualization. 
Hypotheses formation, and empirical simulation of the kind 
finally done by Glass <1972> on the robustness or ANOVA to 
violation of its many assumptions, the results of which 
suprised everyone, including Glass. And the surprise was a 
very pleasant one, particularly for practicing researchers. 

Similarly, Mainer <1989) concluded that the b i crs question 
and area is currently shrouded in a very thick fog, and what 
is needed is not more statistical equations, formulae, and so 
forth, but better conceptualizations, analyses, modeling ana 
thinking about the problem that will indicate some directions 
that eventually will put a dent in one of the most important 
and perplexing methodological problems in research today: 
missing observations. 

This paper, therefore, is an effort in the direction and 
along the lines that Hainer, Angoff, and others have recently 
suggested, although our views of the questions and problems 
were developed independently and in parallel to the work of 
these theorists and in many respects are very different from 
the views and analyses of these theorists whose work, 
thinking, ideas, conjectures, <and courage) we respect and 
admire greatly. Consequently, in this paper, we are going to 



try to outline our views, analyses, and thinking about this 
most fundamental of problems as well as the prototype models 
and ideas that we have, and have explored to date, as well as 
some of the data that support our views and the kind of 
activities and studies that need to be done to advance 
understanding and models in this area and support or deny 
several of our conjectures which are derived from newer 
theories and our experiences. Our primary goals in this 
paper are <1> to share our thinking and work in this area 
with others, <2> to get others interested in the ideas, work 
and issues we present, and <3> to encourage and stimulatv^ 
others to pursue these ideas, models, issues, and conjectures 
independently as no single effort or work is going to be 
definitive or conclusive. There are simply too many 
questions, and too many pieces to the puzzle at this point, 
and not enough data, and particularly simulation data and 
stud i es . 

Initially, we must begin by saying that we are quite 
confident in the essential correctness of the views, 
theoretical analyses, points, criticisms, models, and 
conjectures outlined in this paper within the limits, 
conditions, and boundaries of the particular situations and 
phenomena we are focusing upon in this paper; namely, large, 
con«:ilex, and dynamic "popu ! at i ons" that are sampled in an 
attempt to make (probabilistically) valid statements about 
the population based on the fragment in hand <i,e., the 
actual sample data). However, based upon the sound advice of 
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a highly valued anonymous reviewer of a summary of this 
paper, we wish to emphatically state at this point that our 
work, models, data, and thinking cannot be currently 
construed at this point in their development as definitive 
proof by anyone that their "low response rate" sample is in 
fact a random sample of the population in question, and there 
is in fact nothing "wrong" with their sample and that they do 
not have to be careful and cautious in analyzing it or 
interpreting their results. As will be seen In this paper, 
our position, emphatically, is that the resulting randomness 
or non-randomness of any particular sample in fact is a 
function of several reasonably welt-defined factors and 
events that must be present and operational in the situation, 
and that the conclusion that the actual sample is essentially 
a random or non-random one can only be done probabilistically 
at this point in time, until the appropriate simulation 
studies are done. Our views nay differ, and even differ 
radically from those of others; however, this difference in 
no way implies, or should be taken to imply, that we do nut 
strongly adhere to the purposefully conservative position of 
science as a process with respect to emerging science. 

The views presented in this paper began from two 
extremely simple <but highly anomalous) observations and/or 
experiences. The first of these two observations or puzzling 
experiences was trying to explain how human beings could draw 



valid inferences, make valid deductions, and draw valid 
conclusions consistently, and in a highly reliable fashion, 
from the seemingly highly biased, unrepresentative and very 
small samples (fragments) of reality that they experience 
daily and unquestionably in the real world. This very 
simple, basic, and elementary fact replicated millions of 
times an hour (at a minimum), every hour of every day world 
wide, cannot be explained in any simple, easy, or straight 
forward way by classical sampling and statistical theory, and 
places classical sampling and statistical theory and all of 
the concepts and conceptual framework associated with it in 
the same place and in the same logical and empirical position 
as classical neurobiology was in the fifties when Lashley 
said that he would logically be forced to conclude that 
learning did not and could not exist from the framework, 
viewpoints and the "received" theories of contemporary 
neurob i o I ogy . 

This point is nesther a specious or trivial point or 
argument, nor the stretching of a point or example. It is 
the converse of the above simple point that is always and the 
only one emphasized by classical (statistical) theory. The 
"pro" side of the simple point stated above, however, occurs 
with at least equal frequency and is equally correct and 
valid, and therefore must be explained by classical or 
"received " theory as well. One cannot model and explain 
Just "one side" of a coin and say that this "one-sided" coin 
"exists" and is real and that the other side of the coin does 
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not exist and is not real. This approach Ju^t does not pass 
rtiuster i nte I I octual I y or scientifically. If one is gcing to 
play the •'theory game," then one must play it fair and 
square, and not make "all" claims for "oart" models, because 
the very p?irt one has thrown out Just might be the most 
important part- 

The above point and simple conumdrum is a prime example 
of the sej^ectj^ve attention problem, which is one of the mo«t 
key and fundamental problems currently impeding progress on 
and resolution of the many issues and disputes associated 
with the subject matter of this paper. Selective attention 
is often quietly, subtfely, and fatally biased becauEfe it is 
unweighted and is not paying attention to and open to both 
sides of the coin s e mu I tarseous i y , and dynamically adjusting 
perspective (and theory) on the weight of argument and 
evidence (logical, theoretical and empirical When the 
latter is done, the above coin can spontaneously and 
instanteously "Jump" from one side to another right before 
one's eyes Just like one's perception of ambiguous visual (or 
statistical) figurtis. -.h "Jumps" are considered to be 
"catastrophes'" by "either/or" views, but are not catastrophes 
at al I by more inclusive "both/and" views. 

The coin describtaid above is a "paradox, contradiction, 
or conumdrum" coin. It is not the coin one reads about in 
classical statistical theory. The properties, operations, 
and dynamics of the conundrum coin, moreover, are very 
different from the classical coin, because there are opposing 
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<contradictury ) forces and interactions built into the coin 
that are operati?^jj s i f^xu ! taneous I y and are going to "net out** 
at different time points and conditions to one side of the 
coin or the other being "up . " To is the conundrum coin that 
can e)cp!ain how an allegedly biased sample due to non- 
rejsponders can in fact be random^ or so close to random as 
to constitute totally trivial bias at best. It is also the 
conumdrufn coin that can explain how a sample could and can 
"Jump" from being biased to being a random sample under 
various sets of reasonably well specified conditions. The 
formal theory that best represents the conumdrum or paradox 
coin and the exact dynamics and mathematics of its operations 
is Catastrophe theory (Zeeman, 1976), which will be outlined 
in more detail below. 

The conundrum coin and the fact that human beings draw 
correct inferences and deductions from allegedly biased 
sampler cf everyday reality are very key and fundamental 
po i nts and part of the very essence of the prob I em. These 
two points go to the very roots of the problemp both 
conceptually and historically. If a theory, conceptual 
frameworkp view or position cannot explain very simple^ 
basic, elementary and incontrovertible facts and events that 
can be observed daily and anywhere by anyone, and are 
observed daily and everywhere by everyone (and thus large- 
scale, highly controlled experiments are not even really 
neeued), and the use of this theory, conceptual f ramework y 
view, or yi'^sition forces one to logically conclude that these 



incontrovertible facts and events do not exist, and cannot 
existp and everyday reality is to be denied, then they are 
undeniable and iin i qu i vocab I y fatal flaws, paradoxes, and 
contradictions in the theor'% conceptual framework, view, or 
position and they are not minor, or minor blemishes, or small 
po i nts or qu i rks that may be remed ied in various ways • 

The occurrence and existence of the type of point, 
condition, and situation outlined above is almost always 
indicative of very deep seated and basically fatal flaws, 
errors, and contradictions xl'^ep within the inner vital core 
kernel of the theory, conceptual framework, view, or 
position. There are many, many examples of this point 
historically. Behaviorist theories of language being 
completely unable to explain the production or comprehension 
of a new sentence by an individual, or the comprehension of 
or laughter at a Joke, is one pertinent example. Classical 
learning theories not carrying ovor to the natural learning 
context of the everyday world and in fact "declining and 
falling ( see* McKeach i e , 1979)" is another. 

All of these points are made to quickly, succicntly, and 
simply convey the view that the positions to be presented in 
this paper have a very strong a BCiSEi credibility (warrants) 
basis in simple every day observations, facts, and logic that 
cannot be denied or dismissed or patched over and conce^aled, 
and to emphasize that there is another extremely important 
reason to pursue the concepts, ideas, new theories, and 
thinking directions outlined in this paper beyond sampling 
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theory, research^ and statistics. Since the ideas, concepts 
and new theories outlined in this paper are capable of 
explaining the very simple observation described above, 
working some of these problems, issues, and ideas out could 
make an incredibly valuable contribution to cognitive 
psychology and machine intelligence. So there is an 
additional and highly valuable incentive and prize and this 
prize \g basically why we are so interested in this set of 
interrelated problems. If any of the ideas, points, 
suggestions, and conjectures outlined in this paper prove to 
be essentially correct upon further empirical investigation, 
such findings would be important to many areas other than 
statistics and research methodology, and, this is why we 
believe, like Mainer (1989), that this "nuisance variable" is 
extremely important and worthy of intense study. This point 
is also the reason why we believe that work on this subject 
and this "nuisance variable" problem needs to be highly 
interdisciplinary. Concepts and theories from several 
different disciplines are needed to start making a "real 
conceptual dent in this problem , which according to Mainer 
(1989) is our first re^^l order of business. Consequently, we 
would like to begin with what we believe is one of the 
essent i a I i ngred i ents to prov i d i ng some answers to the many 
conumdrums in this area; namely, Catstrophe theory. 
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Qsiastrofihe Thgory 

Catastrophe theory (Zeeman, 1976> is essentially a 
theory and mathemat ics of non^-^linear functions and 
relationships that is far more specific and precise than 
Chaos theory (Gleiky 1987), which is a general and alt 
emcompasB I ng theory of non--!inear phenomena. Non-linearity 
is a mathematical and theoretical "catastrophe"' for classical 
linear models and conceptualizations of phenomena. A 
theoretical or observational "catastrophe" occurs when the 
linear model predicts the "next data point" or ^esults from 
previous data points and the predicted result in reality is 
observed to be in exactly the logically and mathemat i ca I! y 
opposite place. 

Seven very distinct and different general "catastrophe" 
models have been developed (Zeeman, 1976) which range from 
simple to complex depending on the number of observable 
factors and behavioral dimensions that are related together 
in the model. Each of these general models describes how the 
"regression surface" or "plane'' in the linear view is 
"folded" into contoured shapes or structures that can be 
described by mathematical equations which make the data 
orderly, structured, and predictable, both theoretically and 
logically, as well as mathematically. 

The key feature of and fundamental principal embedded 
within each of these seven specific Catastrophe theory models 
is that event points jump from one '"logic^il" state to the 
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logical inverse of the preceding "logical'* state due to the 
interactions of and buildup <net> effects of reciprocal and 
contradictory factors operating sifgultanefiugly upon the point 
in question. The more interacting factors affecting the 
pointy the more complicated the full event space model and 
its folding is, but as the number of contradictory factors 
operating increase, they tend to net out to zero net effects, 
except for very specific and well-defined regions within the 
event space of the model where "jumps" become "confined-" 
This later point is a very important point for a number of 
reasons, but most particularly in terms of the selective 
attention problem (focusing on one factor and the effects of 
one factor only), and the net effects of many contradictory 
or opposing factors interacting and operating simultaneously 
upon an event or event point netting out to zero effects 
(namely, no resulting bias). 

Catastrophe theory is a very specific and very precise 
theory and mathematics of interaction and non-linear 
phenomena, which is to say conditional, phenomena, rather than 
general, simple and uniform phenomena. Catastrophe theory is 
not only the other side of the conumdrum coin, but it can in 
fact explain the condundrum coin and its operations even to 
predicting when the conundrum coin will land on its side. It 
should be noted, however, that Catastrophe theory^s models of 
interactions are very specific, very precise, and well 
bounded models of interactions that are unlike those of ANOVA 
which are general and almost completely unbounded and do not 
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have theory contradicting states or outcomes like Catastrophe 
theory does. Catastrophe theory, moreover, is also far more 
precise and specific than chaos theory, or fractal theory 
<Mandrebot, 1975>, which also suggests that samples from 
large, comple}^, and dynamic "populations" that are influenced 
by many contradictory factors operating simultaneously will 
be extremely good "exemplars" of that "population." These 
implications of fractal theory, which address the feature of 
and the implications and effects of the structure, structural 
comp I ex i ty , and the structural effects of a "population" on 
consequent and result samples (fragments of the population) 
are incredible ideas that are at first so counter- i ntu i t i ve 
as to warrant detailed consideration and investigation in 
and of themselves. 

Catastrophe theory is not so mysterious at its simplest 
and most fundamental level. The deep implications of 
Catstrophe theory are that events (samples) can Jump from 
being biased (or ordered) to being random, or vice-versa, 
depending upon the nature of the factors, conditions, and 
interactions simultaneously operating in a given event space^ 
This fundamental point and observation is essentially the 
fundamental point an<! observation of Chaos theory and Fractal 
theory. Both of these later theories, however, point out that 
what may be ordered at one level may be unordered (random) at 
another and v i ce-versa . It is our view, however, that on I y 
Castrophe theory at this point in time has the precision 
neccesary for simulation studies and that is why we h^^ve a 
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strong proforenco for Catstrophe theory as opposed to other 
theories and models. 

Nhat makes catastrophe theory so unique and powerf u I i s 
that it can handle logically contradictory criteria or 
d imens i ons w i th equa I ease and f ac i 11 ty as logically 
consistent or uniform criteria, and it is this feature of 
catastrophe theory that makes it so i n teres t i ng j, and 
interesting in terms of the uniquely different surfaces and 
mode I s that it generates depending upon the number and 
logical type of the input criteria. It is also this feature 
of catastrophe theory that makes it so powerful and capable 
of modeling with great precision such con trad i ctory , 
d i scont i nuous y and non-linear phenomena as aggression, 
anorexia nervosa, the stock market, cathartic release from 
self-pity, the buckling of an elastic beam, phase 
transitions, and a wide variety of other such non-1 inear 
phenomena. Catastrophe theory alone, however, cannot explain 
our data. Fuzzy subset theory (Kaufman, 1979) is a I so needed 
as we I ! as several other concepts. Fuzzy subset theory is 
needed not only to describe the difference between the 
process of sampling real world (fuzzy) populations in the 
real (fuzzy) world, as oppose to the classical ball and urn 
model of these events, but also to clarify the distinction 
that must be made between the nomj^naj^ population and the 
s££s£ilye population (true members who are truly available to 
be sampled and/or respond), particularly relative to large, 
complex, dynamic, and real world populations. However, it is 
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best at this point to discuss our data and data from some 
other researchers, as Catastrophe theory's main ideas of the 
net effects of interacting (contradictory) factors and 
"logical state Jumps" are key ideas and concepts relative to 
the initial discussion and explanation of findi ngs . 

IhS §9.££ild Qk§.®rvat j_on 

Our second extremely simple observation and/or puzzling 
experience occurred when we conducted 3 consecutive annual, 
state-wide, large-scale surveys of graduates from 15 
community colleges and eventually their employers. Through 
computerized records given to us on computer tape by each of 
the 15 community oollegesy we constructed close to a 40 
variable database <N=5,781) on various characteristics of the 
entire (hypothetical) graduate population (see Carifio and 
Shwedel, 1983). Mhen we found that the background 
characteristics of the 20% to 30% of the graduates who 
responded to the surveys were mu 1 1 i var i ate I y no different 
than those who did not respond, we were extreme ly surprised 
and tried to develop some initial conjectures as to why. 
Rut when the same results happened a second time (population 
N-5,627) and a follow-up study of a 10% sample of non- 
responders revealed no differences on any of the 15 dependent 
variables examined, we did more than conjecture, we started 
to build a prototype conceptual model that could explain why 
this could happen (see Carifio et al., 1988). 

Mhen the same results occured a third time (population 
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N=5,129> in a follow up of employers of graduai^es^ we became 
more confident in our thinking and model (see Carifio et al., 
1988) I and then when we encoun'tered °bwo 5'fca'te~wid8 surveys of 
practicing nurses that began with a population database cf 
subjects characteristics of N greater than 5,000 and found 
that 10% samples that had only 40% to 50% response rates in 
terms of the samples survey (making them 5% samples of the 
population) were random (see Fazekas, 1989, and Hunt, 1989), 
we became even more confident in our thinking, prototype 
model and views, and that our results were not Just highly 
improbable flukes. Me also began to have a great deal of 
confidence in older reports of simitar experiences shared 
with us by other researchers engaged in similar activities in 
others states (e.g., Boakes, 1981, and Paulson, 1982). 

The 5 studies described above all have a large number of 
features in common that are extreme I y important to the 
subject of this paper and in explaining the outcomes observed 
in these 5 studies. First, all 5 studies were dealing with 
relatively large and structurally complex "populations'* and 
study spaces as compared to the typical study, but most 
particularly as compared to the classic "ball and urn simple 
experiment model" and/or classical "extensions of this model 
to more complex studies and spaces <see Coombs, 1964 for the 
most detailed of descriptions of these two points). Next, 
both the elements <or members) of these complex populations, 
and a number of complex intervening factors, were distributed 
across the event spaces of these studies, making the event 
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space and the population it contained in each of these 
studies very fuzzy, lumpy, and distributed in effects rather 
than smooth , homogeneous , exact, precise, and certain like 
the classical event and study space model a? is best outlined 
and described by Coombs <1964>. 

In all of the 5 studies described above, the nominal 
population was known in all of its multivariate complexity. 
A complex and complete database existed on both the 
multivariate features of the nomin>^l population and the 
various possible intervening factors and "breakdowns and 
influencing factors" within it. Sample results could be 
checked back against the population mu 1 1 i var i ate I y in terms 
of assessing the degree of bias or randomness of the 
resulting responder sample. This key feature protects and 
protected these five studies from the selective attention <to 
armchair identified) factors problem which is the key feature 
and major flaw in classical theory and classical "views" and 
studies of bias, sampling and the non-responder problem. 
This key weakness is the logical equivalent of the family- 
wise error rate problem in statistical testing and the need 
for control ot this problem in doing statistical tests. This 
key weakness is one of the major reasons why all of the 
classical research, conceptualizations and thinking in this 
area is so severely flawed, highly questionable, and 
completely dubious. We will elaborate this point more fully 
at the end of this paper. 
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All 5 of the studies outlined above had a real world 
fuzzy and distributed (random) variable embedded in the core 
of the conduct of each study that the urn of the classical 
model and hermetic laboratory context does not allow to 
operate or exist. Each study had to construct a definition 
of member and then sieved elements to find the population 
<lf one thinks that It is extremely easy to define a graduate 
of a community college, then one needs to spend some time 
talking to registrars at a few colleges). Next, each study 
had highly dynamic "true members" of the population fitting 
the constructed membership definitions. 

Current street addresses (the major member locator 
variable in the large, distributed, fuzzy urn) were 
Inaccurate for up to 20% of the cases in some instances with 
no discernible pattern (bias) in terms of inaccurate cases. 
This inaccuracy rate in current addresses is not particularly 
high, according to a wide variety of people to whom we have 
spoken who work this type of data and problem. Educated 
professionals are a highly mobile and dynamic population, and 
only an instance of the many kinds of dynamic populations 
there are in all areas. 

We found that in following up on non-responders that a 
goodly number had never received the survey we sent them 
and/or we had not received a goodly number of the surveys 
they had returned. Again, this was another dynamic variable 
that had no di scorn ibie pattern (bias) to it in terms of the 
background variables in our database. Again, in talking to 
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many other people in a variety of different areas, we found 
that this state of affair?^ ig not at all uncommon, and has 
been in fact occurring more frequently in the last few years. 

The effects of all of the above variables in all 5 of 
these studies were <1> to reduce the size of the nominal 
response rates to between 5% and 25%, when in fact the actual 
effective response rates, when all of these factors were 
taken into account, were between 35% and 55% of the effective 
population (see Carifio et al, 1987 for details). The 
responder samples, therefore, were actually very much larger 
samples than one would believe they were, if one evaluated 
response rates in terms of the nominal popluation "urn." The 
net effects of all of the random and contradictory response 
affecting factors in these 5 studies were statistically zero. 

Each of the responder samples in each of the 5 studies 
were de facto statistically random relative to their 
populations, although a wide variety of armchair arguments 
could be made about various (selective) factors that would 
strongly suggest, if not prove, that the responder samples 
were most probably biased samples, when Ihe exact opposite 
was true statistically and de facto. Catastrophe theory, in 
particular, but fuzzy subset theory and Chaos theory also, 
would have predicted, however, that the responder samples in 
these 5 stud i es were most probab I y random and not b i ased . 
These theories, moreover, led to the most simple initial 
formulation of our views. 
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This formulation says that as the structural comploxity 
of a population increases in terms of the number of variables 
characterizing the elements of the population simultaneously, 
and as the number of Interactj^nfl factors increase that 
simultaneously influence the occurence or non-^occurence of 
responding (and as these factors increase in number as 
contradictory factors), then the greater the probability that 
the resulting responder sample will be a random sample of the 
popu I at i on . 

Further, this view says that the probability that the 
resulting responder sample will be a random sample of the 
population is far, far greater (and closer to 1> than the 
probability that the responder sample will not be random; 
namely, than the probability that it will be a sj^gnlf icantiy 
(as opposed to merely statistically) biased sample. This 
very simple and somewhat obvious question of Just how biased 
any given sample is, and exactly how biased is biased, both 
quantitatively and qualitatively, is one of the key, 
critical, and outstanding unanswered questions in this whole 
area, which we will comment on further below. 

We initially thought that this view and our predictions 
could be construed as a restatement of the "Law of Large 
Numbers," where the word "Number" is replaced by the words 
"Number of (interacting) factors involved and number of 
(interacting) events involved in a given study space." It 
could be said that our points and views are inherent, but 
masked, in the classical view as unseen, but inherently 
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present, implications and deductions from the Law of Large 
Numbers. This partictlar point does have some validity to 
it, but it would mean that we would now have to start 
considering and talking about the "Laws of Large Number," 
rather than Just the Law of Large Numbers- We have no real 
problem with such an approach, other than the fact that it 
has taken so long to perceive, and that it could keep us all 
shackled to the emp i r i ca I -descr i pt i ve rather than making the 
transition to the conceptual, theoretical, and enplanatory. 

Stated in the most simple of terms, our view says that 
the number and nature of factors simultaneously operating in 
an event <NNFSO> is as iioeortant as N (sample sire) in 
determining the resulting state of the event (i.e., the 
nature of the resulting sample). This point, it should be 
noted, is also the implication of Chaos theory. Fractal 
theory, and fuzzy subset theory, but it is only Catastrophe 
theory that gives the simplest, clearest, and most precise 
explanation as to why this is and will be so. 

The simple Cusp catastrophe theory model, wh i :?h is 
essentially a bimodal state probability distribution, 
predicts that the sample of responders under the types of 
conditions described above will gradually begin to shift and 
then eventually "instantaneously Jump" from being a biased 
sample to being a random sample of responders as the sample 
size increases, and the number of factors and contradictory 
factors affecting responding increase in number, and then 
S±a^ E§Qd2!D» to the effects of all of these interacting 
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factors netting out to zero, oarticularly for large sample 
sizes. This simple model and the various factors identified 
above both explain and account for the observed resulting 
random samples in the 5 studies described. Catastrophe 
theory's predictions are very precise^ specific, and 
simulatable. Catastrophe theory says that depending upon the 
nature of the factors operating In a given study space (NNFSO 
and N>, one of two "strange attractors" will begin to operate 
and prevail eventually in the study space. The first of 
these two "strange attractor" is bias (which has been 
somewhat cursorily studied by the classical view), and the 
second strange attractor is randomness, which will occur and 
prevail in large, complex, and dynamic populations, whose 
samples are of some reasonably sufficient percentage size. 
It is this second "strange attractor" that is and has been 
the focus of this paper. 

It should be noted that "strange attractors" are most 
often asBociated with "scale d i scont i nu i t i es, " and structural 
discontinuities between the "simple" and the "complex^" 
The populations and samples in the 5 studies described above 
were not large and complex in terms of our meaning of the 
terms large qnd complex, but rather only sufficiently large 
and complex to cross the threshold into the domain of the 
large and complex that is the focus of this paper. The 
populations in the studies reported above were only large and 
comp lex as co.^pared to typ i ca I stud i es where the popu I at i on 
and sample N's are usually 1/5 to 1/10 the size. However, i b 
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is a fairly well documented fact that when an effect begins 
to manifest itself in a st* mistical model, it starts abruptly 
and then moves quickly, and we only have to plot values on 
look-up tables in statistical texts to demonstrate this 
continually recurring fact. The interesting observation is 
that only Catastrophe theory is really capable of giving a 
very full and detailed explanation of this abrupt "kick-in 
effect," which typically tends to be depicted as simply an 
emp i r i cal -descr i pt i ve fact. Empirical descriptions are not 
explanations and empirical descriptions are not theories. 

Strange attractors are a key concept of Chaos theory, 
but they are also inherent and most definitely present in all 
seven Catstrophe theory models, each of which describes and 
models, both conceptually and mathematically, the general 
points, conditions, and areas where "strange attractors" will 
develop and occur. It is the concrete and specific character 
of Catastrophe theory that has drawn us to it as a model and 
as a thinking model for all of these puzzling questionc. 
Also, Castrophe theory has a specific and clear "semantic" 
and explanatory component to it which these other theories do 
not. 

Qihsr Efi§iyrss 

There are several other features of the above studies 
that need to be noted relative to the remainder of the 
discussion in this paper. First, the operational definition 
of what constitutes a "ball" in an "urn" in a model of a 



ERIC 



24 2 7 



ft 



simple experiment is rarely a complex, let alone a fuzzy, 
definition. "Balls" are not structurally complex entities in 
the classical model, and neither are the typical populations 
of "balls" studied in "sampling simulations." Next, in 
conducting a "ball and urn simulation," one is not making a 
large number of decisions, nor does one have a distributed 
dS£i§.±2!lz!D§!<i!ia !D2^Si operating where many different 
individuals <or Selfridge's demons) are making decisions in 
parallel producing a net result, outcome, or effect (see 
Zadeh, 1974b and Caudill, 1990). 

This distributed decision-making model (and its effects) 
is operating at every step in the process in the event spaces 
of the 5 studies described above. The more elements (and/or 
demons) participating in a distributed decision-making model, 
and the more often that th i s mode I is ope rat i ng aga in and 
again in a sequential process, the greater the probability 
that the errors produced by this model will be random, 
because the probability that a particular bias consistently 
and consecutively operate effectively and homogeneously this 
many times in a row is very, very, very close to zero. This 
later point is also the "tails side" of the conundrum coin« 

This distributed decision-making ;iiodel, and its not 
effects, are operating at every step in the "real time 
conducting of the study processes" in the 5 studies described 
above to produce net effects which interact with every prior 
and subsequent net effect. As the number of factors in this 
"real time" model increase, and as the number of factors 
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become contradictory, the net effects across the model will 
be randomness, particularly at the higher levels of 
aggregation and data-breaks. One of the major factors that 
affects the "contradiction index or quotient" of the factors 
operating in the real-time model is the (relevant) structural 
complexity of the traits of the elements that constitute and 
comprise the population. 

As a population, and the samp ling, research p recess, and 
design become more heterogeneous and "complex" in terms of 
the number of traits and the number of factors involved and 
examined, and their interrelations and interactions, the 
greater the probability that the resulting responder sample 
will move towards randomness, due to the net effects of this 
complexity. Therefore, the resulting sample will be less 
biased, at a minumum, than we would think, and particularly 
think from the viewpoint of classical theory. Consequently, 
our data would be much better than our (classical) beliefs 
about our data and we would be highly prone to making "beta 
errors" and also in actually making them in the analysis and 
interpretation of the data. 

All of the above points say that the data from surveys 
of I arge , comp lex, and dynam i c popu I at i ons are most probab I y 
far, far better than we believe this data to be. Otherwise, 
how would we truly <on the average) get along and survive in 
the everyday world. All of these same points would also hold 
for experiments that fit this large,, complex, and dynamic 
moiel, and all of the points that we have made also have a 
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variety of implications relative to meta-ana I ys i s • There 
arOf however^ also several other reasons to believe and 
pursue all of the points that have been outlined above, 

EafflilyzHisfi Errsr Bates 

He know that in a complen set of data or a complex study 
space^ we can find significant differences <by chance) if we 
selectively focus on a single and only single feature of the 
complexity. We know that this finding is a false or 
*'phantom" finding because we have not analyzed the complexity 
mu 1 1 i var i ate I y and controlled the family-wise error rate in 
our analyses* The family-wise error rate problem is the 
selective attention problem in a nutshell^ and the perfect 
*^quick exemplar*' of our points and our points about bias and 
the biased sample problem. The family wise error rate 
problem is an exemplar of the conundrum coin and how it 
'^spontaneous I y flips," and why both sides of the coin must be 
attended to simultaneously by a balanced algorthm and 
sequential decision making process <see Zadeh, 1974a). One's 
attentional and analytical model must be full and open 
Initially and then sequentially move through a "controlled'' 
process examining for and against data and arguments to 
determine which side of the coin should be up or if the coin 
should be standing on its edge. It is the clarification and 
simulation of this process, moroever, that is the greatest of 
ou tstand ing research needs. 
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Uo do not even have rough gauge empirical rules of thumb 
when it comes to the bias question in any shape or form, and 
this is why simulation studies are so strongly needed in this 
area. And the type of simulation studies that need to be 
done are not the scrupulously clean and overly simplistic in 
the extreme "ball and urn" classical model studies with one 
and on I y one (extreme I y gross > b e as i ng factor operat i ng and 
affect i ng seu I ts. 

He really do not qualitatively and precisely know how 
much any given '^biasing" factor or set of factors biases a 
given sample whose population has a particular well-defined 
set of characteristics. There is no metric, not even a broad 
gauge relational metric of the orders of magnitude kind to 
assess any study or any discourse or claim on the bias 
question in any form. The significant question and 
significant research question is not biased or unbiased in 
terms of results, but how biased; in the second decimal 
place, the first digit, a tenth of a standard deviation, a 
whole standard deviation, how biased exactly? Answering this 
question to some ordinal degree is the only real way that a 
given sample or study can be rejected or accepted using some 
type of practical and consistent rule of thumb. 

Me are aware of no samp ling s i mu I at i on stud i es that 
real ly answer the question how biased, and/or how biased as a 
function of what factors and/or conditions. Claims of bias 
and biased results spring up like Topsy all over the research 
landscape daily again and again, all typically the results of 
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long inferential and speculative chains of arguments and 
small shreds of suggestive data. When the most major of 
these claims are actucilly tested and checked out in Monte 
Carlo studies, they most often are found to be unfounded or 
grossly exaggerated (e.g., Glass et al., 1972; Dagenais and 
Marasculio, 1973; Richards, 1972). The weight of evidence of 
all of these studies should tell us all something; namely, 
that assertions of alleged bias should be required to be 
proved before they are even entertained, because the results 
of the majority of MHlM experimental bias investigations and 
simulations tend to be negative and the bias is proved to be 
non-existent, trivial, or to hold in only highly restricted 
(and usually extreme) situations. The weight of the somewhat 
extensive and high quality existing research evidence has to 
enter in to the consideration of the bias question somehow , at 
some level, at some point, even only as a rule of thumb to 
evaulate claims and assertions as we work on the problem. 
The really outstanding unanswered question is why it does 
not. 

No one, we believe, can currently state how biased any 
given sample will be under any set of real world conditions, 
and state it in a way that will roughly prove out to be 
correct and support cj[assicai models and vj^ews of all of 
these questions on empirical simulation. All such estimates 
most professionals currently make, we believe, are subjective 
(bayesian) armchair guesses, and subjective (Bayesian) order 
of magnitude estimates in the long standing behaviorist 
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post-hoc and armchair tradition that is the classical core of 
the (overlapping) statistics and research methodology 
disciplines, as viewed and construed by the majority of 
current practitioners. Consequent I y , we have assumed in all 
of our discussions above, and will assume in all of our 
discussion belcw, that wo have a simple "Turing Device" that 
generates an estimate for us of how biased a given resulting 
sample will be under a particular set of conditions. Our 
simple Turing Device, however, is a fuzzy and not exact 
Turing Device. 

Zulu Subset Theory 

Classical sampling and survey theory, in the main, are 
based on classic set theory and linear model views of the 
world, which in turn drives all of their logic, inferences, 
deductions and conclusions (e.g., see Scheffe, 1953; Coombs, 
1964; Hayes, 1981; Cochran, 1983; and Kerlinger, 1986). 
Kaufman's (1976) great insight was that neither classical set 
theory nor its logic could be applied to real world problems 
in any easy, precise, accurate, or truly meaningful way. 
Kaufman, therefore, developed fuzzy subset theory to deal 
with real-life^ set theory, problems, situations, and human 
thinking, statistically. Fuzzy subset theory is a complete 
rev i s i on of c I ass i cal set theory to i ncorporate the concepts 
of uncertainty and probability into set theory. Kaufman, in 
devising fuzzy subset theory, actually revised classical 
sampling theory, although the implications of these revisions 
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were not pursued in any significant way by Kaufman or any 
other statistical theorist^ to the best of our knowledge. 
There is, however, a reasonably well worked out mathematics 
of defining fuzzy subsets, manipulating fuzzy subsets and 
manipulating them statistically. 

In fuzzy subset theory , there are , un I i ke c I ass i ca I sa h 
theory, only subsets (due to the "fuzz"), and every set has 
two basic rules. The first rule is the "olassioal" rule <or 
procedure) that defines membership or non-membership in a 
given subset, and the second is the rule <or procedure) that 
defines the probability of each member in a given subset as 
being a "true" member of the subset and its probability of 
being a member of an alternative subset <or set of subsets). 
The probability of true membership in the logical subsets is 
the "fuzz" and it is the "fuzz" that is of prime importance. 
Unlike the well-defined and hermetic urns, balls, and 
processes of classical set and sampling theory, membership of 
any element in any subset is not a given and absolute, but 
rather fuzzy and probabilistic for some percentage of the 
elements, and this is Kaufman's real world, key insight and 
point. 

The above features of fuzzy subset theory forces a 
distinction to be made between the nonjinai set, subset and/or 
fififiyiaiiSD < which is the classical view), and the effoctlije 
set, subset, and/or ROEuiation, which are the members of the 
set, subset, or population that have the highest 
probabilities of being true members of the set or nominal 
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population in question (which is the fuzzy view). This 
distinction is the key and core concept of this view and it 
is capable of explaining how survey response rates of 10% to 
15% of the nominal populuation can with a very, very high 
probability easily be random and representative samples of 
the population, if the initial samples were randomly drawn, 
particularly for large, complex, and dynamic populations that 
are "fuzzy" in nature, such as the populations in the 5 
studies described above. 

From fuzzy subset theory, it can be shown that in a 
"dynamic" population, there is always an effective subset <or 
population) that has a higher probability of being randomly 
drawn (located, accessed and sucessfully "captured as a 
respondent") than another subset; namely, the effective 
population is an intervening variable that classical sampling 
theory does not take into account because it is based on 
classical set theory and not fuzzy subset theory. This 
"inability to access and capture" condition was present in 
all 5 of the studies described above, and a wide variety of 
factors create large and important differences between the 
nominal and effective populations in these types of survey 
stud i es» 

The question, therefore, becomes is the effective subset 
actually sampled biased or random. The answer depends on two 
factors, both of which were identified and explained in 
detail in the discussion on Castrophe theory above. The 
first of these two factor is whether or the factor creating 
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the effective population is random or not. This "first 
factor," it should be noted, could also be the effects of 
several interacting factors netting out to zero and 
randomness (or not). If this "Influencing factor" \b 
essentially a random factor, then the effective population 
will statistically be a ver;^, ver^ large random sample of the 
nominal population. However, if there is a bias, depending 
on I ts degree of sever i ty , and the size of the ef f ect i ve 
population as a percentage of the nominal population, the 
effective population for large scale populations will be 
ySOj. very close to a very, very large random sample of the 
nominal population. So even with biasing factors, fuzzy 
subset theory predicts that one comes very, very close to a 
random sample at large scales, and this most probable fact is 
a reasonably strict deduction from fuzzy subset theory, and 
only counter — intuitive from the point of view of classical 
set and sampling theory. Further, this prediction is why a 
good metric and scale for "how biased" is so badly needed. 

The second factor that affects whether or not the 
effective population and samples drawn from it will be random 
or not is the number of influencing factors operating In the 
situation. This factor and its operations have been 
previously explained above in detail. When many factors are 
intei .;ting together to create the effective population 
and/or samples drawn from it, the net results are going to 
tend to be random almost all of the time when NNFSO and N are 
large. The reason that the samples will tend to be random is 
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the same as that for the effective population as the same 
principle is operating in the second step of the process. 

From fuzzy subset theory and all of the points made in 
alt of the discussions above, et is not too difficult to see 
why in large scale surveys, response rates are always far, 
far higher than calculated by conventional methods and one 
has d far, far greater sample size <of the effective 
population) than one believes and is represented nominally, 
which means that one's data is much, much better than one 
would believe by traditional views. Also, in large-scale 
surveys of large, complex, and dynamic populations, the odds 
are very high that one's responder sample is a random sample 
of the nominal population when it is 15% to 20% of the 
nominal population. A simulation study could be designed to 
show that this particular outcome would be so most of the 
time. The value in conducting this simulation study would be 
the provision of exact points and cutting ratios rather than 
i. . estimated points and rough rules of thumb given above. 

Fuzzy subset theory strongly suggests that sample data 
of a reasonably sufficient size from large, complex, fuzzy, 
and dynamic populations is probab I i st i cal I y much better than 
we believe it is, according to the classical views; and this 
is good news, even if what the data tell us is not good news. 
People, therefore, may be ignoring and holding suspect 
perfectly good data. The common w i sdom and o I der 
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interpretations prevail, even when newer theory, simple 
logicp intuition, experience, and empirical data give clear 
messages to the contrary. 

iDterifg Conclusions 

We believe that the weight of argument, empirical 
evidence, and newer theories, at this time, is sufficiently 
strong enough to suggest that challenges of sampling bias 
based soley on arguments of low response rates are highly 
suspect and quite dubious at best, but most particularly so 
under the sets of conditions outlined in this paper. Me 
blleve that under the sets of conditions outlined in this 
paper that allegedly biased samples are not as biased as 
those that claim bias believe, and that a great deal of 
reasonably good real world data is being dismissed and 
ignored when it should not be. 

Me have outlined the basis of our views, and studies 
that support our views, as well as numerous studies and 
simulations that need to be done to confirm, partially 
confirm, deny, or partially deny our views. Me believe that 
the onus should be upon those who claim bias, particularly 
under the study conditions we have described, to prove their 
claims or disprove ours as vice-versa, which is the current 
case. It is our view that such claims should no longer be 
unquest i on i ng I y accepted by researchers, or accepted as 
proven fact until the appropriate evidence is generated to 
settle the question. The case against these claims. 
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currently, is stronger than the case for these claims, and 
all newer theory strongly suggests that the clasical views 
and claims on these questions will not be supported or well 
supported when the appropriate studies and simulations are 
done . 

The weight of argument, evidence, and nev^er theories 
(that all converge in the same place) is sufficient, we 
believe, to warrant this view, and to cast very serious 
doubts upon claims of alleged bias based on classical models 
and views. It is really a matter of which new theory or set 
of now theories that one "^ishes to chose to operate from to 
get to the conclusior i wo h^ve drawn and the predictions that 
we have made. Our cho;-;;^^^ Catastrophe theory supplemented 
by fuzzy subset theory, for al I of the reasons we have stated 
In this paper. Catastrophe theory Is the only theory <to 
date) that precisely explains and predicts the conundrum coin 
and its various operations. The very strong suggestion of 
these newer views, if they prove to be essentially correct 
upon empirical simulation, would be a very important 
discovery, not only for the areas of research methodology and 
statistics, but also for the areas of cognitive psychology 
and machine intel I igence. 

Real world data is much, much better than classical 
views say it is, or lead people to believe it is, and this is 
a problem of great importance and a problem that is very 
significant both theoretically and practically. 

Some of U6 mho worked in this world of ''classically 
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suspect data" developed an intuitive feeling about the 
problem and sought out models and theories to formalize our 
intuitions and guide research and thinking, which is still in 
its early stages, but all of the empirical data and the 
theory is supporting our views. And our views and the data 
are saying that we are essentially correct and empirically 
observing what we should be seeing, and that the problem is 
in the "hermetic urn" views which need to be revised, 
rad i ca I I y . 

Classical survey and sampling theory is essentially 
seriously flawed and fundamentally incorrect and misleading 
in several important respects because it has not and does not 
incorporate fuzzy subset. Catastrophe theory and Chaos theory 
principles into its views. At a very minimum, therefore, 
this fact means that we Just might now be more open and more 
accepting of large-scale, real world data rather than Just 
dismissing it due to "flawed methodology," which Just might 
save us in a number of different ways in the long runi. 
Consequently, it is the "downside" problems and flaws of the 
classical views that are the most problematic and flawed 
aspects of the classical views, which is a model or view 
adoption cost that is rarely considered or considered in the 
evaluation of a view, or the decisions and claims that result 
from it. 

As Student pointed the way for statisticians in the 
early part of the century in terms of understanding that the 
"very" small and the finite were different from the "typical 
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laboratory model" of the day that was being used, the newer 
theories outlined in this paper are pointing the way for 
statisticians now in terms of showing that the large, complen 
anri dynamic are very different from the "typical laboratory 
model" that is being used today. The large, fuzzy, dynamic 
and complex, and the small, well-defined and simple are very, 
very different from each other. He all need to begin to 
accomodate to and deal wi th th i s bas i c fact. 

Catastrophe, Chaos, and Fuzzy Subset theory as theory 
names does not mean, or in any way imply, unpredictability or 
i ncomprehens i b i ty . This misconception is one of the most 
common misconceptions of these theories currently. Although 
still under development, these theories are in fact Just the 
opposite of unpredictability and incomprehensibility; namely, 
they lead to high predictability of events, phenomena, 
outcomes, and results, rather than paradox and contradiction. 

There are two kinds of general phenomena in this world 
and not one; namely, uniform and conditional phenomena. 
There are many conditional phenomena in education (learning 
is but one), and these conditional phenomena need to be 
modelled with conditional theories, if we are to understand 
them adequately. Both models and views, therefore, are 
needed and needed in ful ly developed formulations to conduct 
reasonable and good research. Practicing researchers and 
theoribts simply cannot be be expected to or made to play 
with one hand tied behind their backs. A great deal of new 
work is needed and has been needed for a very long time. 
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With just a little bit of thtiory, conceptualization, legwork, 
and study design modification, both the quality and certainty 
of all of large-scale survey data could be improved 
immmensely, as well as the cost-benefit ratios. And this is 
the good new, even if what the data has to say is bad news. 
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